Duplicate path contexts in C# extractor #7

ASvyatkovskiy · 2019-01-07T20:00:53Z

The pull request introduces a uniqueness check to avoid duplicate paths contexts in the C# extractor.
For an example code snippet:

using System;
namespace Test
{
    class Program
    {
        static int TestMethod(int n)
        {
            if (n == 0) return 0;
            if (n == 1) return 1;
            if (n == 2) return 2;

            return -1;
        }
    }
}

I get following duplicate path contexts (adding full span of start/end syntax tokens to distinguish cases with the same token name but different location in the code snippet): [1] Is it expected behavior, or such path-contexts should be removed?

In addition, the PR suggests sampling terminal nodes in Extractor.GetInternalPaths, before extracting the paths connecting them. The suggested sample size is 30000 - would affect only very deep/wide ASTs. Currently the Sampling down to 200 path contexts is performed on the post processing step https://github.com/tech-srl/code2vec/blob/master/preprocess.py#L23.

[1]

[85..89) int,PredefinedType^Parameter^ParameterList^MethodDeclaration_PredefinedType,int [70..74)
 3
[70..74) int,PredefinedType^MethodDeclaration_ParameterList_Parameter_PredefinedType,int [85..89)
 3
[70..74) int,PredefinedType^MethodDeclaration_ParameterList_Parameter,n [89..90)
 8
[85..89) int,PredefinedType^Parameter,n [89..90)
 8
[70..74) int,PredefinedType^MethodDeclaration,METHOD_NAME [74..84)
 2
[85..89) int,PredefinedType^Parameter^ParameterList^MethodDeclaration,METHOD_NAME [74..84)
 2
[118..120) n,IdentifierName^EqualsExpression^IfStatement^Block^MethodDeclaration_ParameterList_Parameter,n [89..90)
 10
[152..154) n,IdentifierName^EqualsExpression^IfStatement^Block^MethodDeclaration_ParameterList_Parameter,n [89..90)
 10
[186..188) n,IdentifierName^EqualsExpression^IfStatement^Block^MethodDeclaration_ParameterList_Parameter,n [89..90)
 10
[89..90) n,Parameter^ParameterList^MethodDeclaration_Block_IfStatement_EqualsExpression_IdentifierName,n [118..120)
 10
[152..154) n,IdentifierName^EqualsExpression^IfStatement^Block_IfStatement_EqualsExpression_IdentifierName,n [118..120)
 10
[89..90) n,Parameter^ParameterList^MethodDeclaration_Block_IfStatement_EqualsExpression_IdentifierName,n [152..154)
 10
[118..120) n,IdentifierName^EqualsExpression^IfStatement^Block_IfStatement_EqualsExpression_IdentifierName,n [152..154)
 10
[186..188) n,IdentifierName^EqualsExpression^IfStatement^Block_IfStatement_EqualsExpression_IdentifierName,n [152..154)
 10
[89..90) n,Parameter^ParameterList^MethodDeclaration_Block_IfStatement_EqualsExpression_IdentifierName,n [186..188)
 10
[152..154) n,IdentifierName^EqualsExpression^IfStatement^Block_IfStatement_EqualsExpression_IdentifierName,n [186..188)
 10
[89..90) n,Parameter^ParameterList^MethodDeclaration_Block_IfStatement_EqualsExpression_NumericLiteralExpression,0 [123..124)
 4
[118..120) n,IdentifierName^EqualsExpression_NumericLiteralExpression,0 [123..124)
 4
[152..154) n,IdentifierName^EqualsExpression^IfStatement^Block_IfStatement_EqualsExpression_NumericLiteralExpression,0 [123..124)
 4
[89..90) n,Parameter^ParameterList^MethodDeclaration_Block_IfStatement_ReturnStatement_NumericLiteralExpression,0 [133..134)
 4
[118..120) n,IdentifierName^EqualsExpression^IfStatement_ReturnStatement_NumericLiteralExpression,0 [133..134)
 4
[152..154) n,IdentifierName^EqualsExpression^IfStatement^Block_IfStatement_ReturnStatement_NumericLiteralExpression,0 [133..134)
 4
[89..90) n,Parameter^ParameterList^MethodDeclaration_Block_IfStatement_EqualsExpression_NumericLiteralExpression,1 [157..158)
 10
[118..120) n,IdentifierName^EqualsExpression^IfStatement^Block_IfStatement_EqualsExpression_NumericLiteralExpression,1 [157..158)
 10
[152..154) n,IdentifierName^EqualsExpression_NumericLiteralExpression,1 [157..158)
 10
[186..188) n,IdentifierName^EqualsExpression^IfStatement^Block_IfStatement_EqualsExpression_NumericLiteralExpression,1 [157..158)
 10
[89..90) n,Parameter^ParameterList^MethodDeclaration_Block_IfStatement_ReturnStatement_NumericLiteralExpression,1 [167..168)
 10
[118..120) n,IdentifierName^EqualsExpression^IfStatement^Block_IfStatement_ReturnStatement_NumericLiteralExpression,1 [167..168)
 10
[152..154) n,IdentifierName^EqualsExpression^IfStatement_ReturnStatement_NumericLiteralExpression,1 [167..168)
 10
[186..188) n,IdentifierName^EqualsExpression^IfStatement^Block_IfStatement_ReturnStatement_NumericLiteralExpression,1 [167..168)
 10
[89..90) n,Parameter^ParameterList^MethodDeclaration_Block_ReturnStatement_UnaryMinusExpression_NumericLiteralExpression,1 [225..226)
 10
[186..188) n,IdentifierName^EqualsExpression^IfStatement^Block_ReturnStatement_UnaryMinusExpression_NumericLiteralExpression,1 [225..226)
 10
[89..90) n,Parameter^ParameterList^MethodDeclaration_Block_IfStatement_EqualsExpression_NumericLiteralExpression,2 [191..192)
 8
[152..154) n,IdentifierName^EqualsExpression^IfStatement^Block_IfStatement_EqualsExpression_NumericLiteralExpression,2 [191..192)
 8
[186..188) n,IdentifierName^EqualsExpression_NumericLiteralExpression,2 [191..192)
 8
[89..90) n,Parameter^ParameterList^MethodDeclaration_Block_IfStatement_ReturnStatement_NumericLiteralExpression,2 [201..202)
 8
[152..154) n,IdentifierName^EqualsExpression^IfStatement^Block_IfStatement_ReturnStatement_NumericLiteralExpression,2 [201..202)
 8
[186..188) n,IdentifierName^EqualsExpression^IfStatement_ReturnStatement_NumericLiteralExpression,2 [201..202)
 8
[89..90) n,Parameter^ParameterList^MethodDeclaration,METHOD_NAME [74..84)
 4
[118..120) n,IdentifierName^EqualsExpression^IfStatement^Block^MethodDeclaration,METHOD_NAME [74..84)
 4
[152..154) n,IdentifierName^EqualsExpression^IfStatement^Block^MethodDeclaration,METHOD_NAME [74..84)
 4
[186..188) n,IdentifierName^EqualsExpression^IfStatement^Block^MethodDeclaration,METHOD_NAME [74..84)
 4
[133..134) 0,NumericLiteralExpression^ReturnStatement^IfStatement_EqualsExpression_NumericLiteralExpression,0 [123..124)
 3
[123..124) 0,NumericLiteralExpression^EqualsExpression^IfStatement_ReturnStatement_NumericLiteralExpression,0 [133..134)
 3
[123..124) 0,NumericLiteralExpression^EqualsExpression^IfStatement^Block^MethodDeclaration_ParameterList_Parameter,n [89..90)
 4
[133..134) 0,NumericLiteralExpression^ReturnStatement^IfStatement^Block^MethodDeclaration_ParameterList_Parameter,n [89..90)
 4
[123..124) 0,NumericLiteralExpression^EqualsExpression_IdentifierName,n [118..120)
 4
[133..134) 0,NumericLiteralExpression^ReturnStatement^IfStatement_EqualsExpression_IdentifierName,n [118..120)
 4
[123..124) 0,NumericLiteralExpression^EqualsExpression^IfStatement^Block_IfStatement_EqualsExpression_IdentifierName,n [152..154)
 4
[133..134) 0,NumericLiteralExpression^ReturnStatement^IfStatement^Block_IfStatement_EqualsExpression_IdentifierName,n [152..154)
 4
[123..124) 0,NumericLiteralExpression^EqualsExpression^IfStatement^Block_IfStatement_EqualsExpression_NumericLiteralExpression,1 [157..158)
 6
[133..134) 0,NumericLiteralExpression^ReturnStatement^IfStatement^Block_IfStatement_EqualsExpression_NumericLiteralExpression,1 [157..158)
 6
[123..124) 0,NumericLiteralExpression^EqualsExpression^IfStatement^Block_IfStatement_ReturnStatement_NumericLiteralExpression,1 [167..168)
 6
[133..134) 0,NumericLiteralExpression^ReturnStatement^IfStatement^Block_IfStatement_ReturnStatement_NumericLiteralExpression,1 [167..168)
 6
[123..124) 0,NumericLiteralExpression^EqualsExpression^IfStatement^Block^MethodDeclaration,METHOD_NAME [74..84)
 2
[133..134) 0,NumericLiteralExpression^ReturnStatement^IfStatement^Block^MethodDeclaration,METHOD_NAME [74..84)
 2
[167..168) 1,NumericLiteralExpression^ReturnStatement^IfStatement_EqualsExpression_NumericLiteralExpression,1 [157..158)
 6
[157..158) 1,NumericLiteralExpression^EqualsExpression^IfStatement_ReturnStatement_NumericLiteralExpression,1 [167..168)
 6
[157..158) 1,NumericLiteralExpression^EqualsExpression^IfStatement^Block^MethodDeclaration_ParameterList_Parameter,n [89..90)
 2
[167..168) 1,NumericLiteralExpression^ReturnStatement^IfStatement^Block^MethodDeclaration_ParameterList_Parameter,n [89..90)
 2
[225..226) 1,NumericLiteralExpression^UnaryMinusExpression^ReturnStatement^Block^MethodDeclaration_ParameterList_Parameter,n [89..90)
 2
[157..158) 1,NumericLiteralExpression^EqualsExpression^IfStatement^Block_IfStatement_EqualsExpression_IdentifierName,n [118..120)
 2
[167..168) 1,NumericLiteralExpression^ReturnStatement^IfStatement^Block_IfStatement_EqualsExpression_IdentifierName,n [118..120)
 2
[157..158) 1,NumericLiteralExpression^EqualsExpression_IdentifierName,n [152..154)
 2
[167..168) 1,NumericLiteralExpression^ReturnStatement^IfStatement_EqualsExpression_IdentifierName,n [152..154)
 2
[157..158) 1,NumericLiteralExpression^EqualsExpression^IfStatement^Block_IfStatement_EqualsExpression_IdentifierName,n [186..188)
 2
[167..168) 1,NumericLiteralExpression^ReturnStatement^IfStatement^Block_IfStatement_EqualsExpression_IdentifierName,n [186..188)
 2
[225..226) 1,NumericLiteralExpression^UnaryMinusExpression^ReturnStatement^Block_IfStatement_EqualsExpression_IdentifierName,n [186..188)
 2
[157..158) 1,NumericLiteralExpression^EqualsExpression^IfStatement^Block_IfStatement_EqualsExpression_NumericLiteralExpression,2 [191..192)
 4
[167..168) 1,NumericLiteralExpression^ReturnStatement^IfStatement^Block_IfStatement_EqualsExpression_NumericLiteralExpression,2 [191..192)
 4
[225..226) 1,NumericLiteralExpression^UnaryMinusExpression^ReturnStatement^Block_IfStatement_EqualsExpression_NumericLiteralExpression,2 [191..192)
 4
[157..158) 1,NumericLiteralExpression^EqualsExpression^IfStatement^Block_IfStatement_ReturnStatement_NumericLiteralExpression,2 [201..202)
 4
[167..168) 1,NumericLiteralExpression^ReturnStatement^IfStatement^Block_IfStatement_ReturnStatement_NumericLiteralExpression,2 [201..202)
 4
[225..226) 1,NumericLiteralExpression^UnaryMinusExpression^ReturnStatement^Block_IfStatement_ReturnStatement_NumericLiteralExpression,2 [201..202)
 4
[157..158) 1,NumericLiteralExpression^EqualsExpression^IfStatement^Block^MethodDeclaration,METHOD_NAME [74..84)
 3
[167..168) 1,NumericLiteralExpression^ReturnStatement^IfStatement^Block^MethodDeclaration,METHOD_NAME [74..84)
 3
[225..226) 1,NumericLiteralExpression^UnaryMinusExpression^ReturnStatement^Block^MethodDeclaration,METHOD_NAME [74..84)
 3
[201..202) 2,NumericLiteralExpression^ReturnStatement^IfStatement_EqualsExpression_NumericLiteralExpression,2 [191..192)
 3
[191..192) 2,NumericLiteralExpression^EqualsExpression^IfStatement_ReturnStatement_NumericLiteralExpression,2 [201..202)
 3
[191..192) 2,NumericLiteralExpression^EqualsExpression^IfStatement^Block_IfStatement_EqualsExpression_NumericLiteralExpression,1 [157..158)
 2
[201..202) 2,NumericLiteralExpression^ReturnStatement^IfStatement^Block_IfStatement_EqualsExpression_NumericLiteralExpression,1 [157..158)
 2
[191..192) 2,NumericLiteralExpression^EqualsExpression^IfStatement^Block_IfStatement_ReturnStatement_NumericLiteralExpression,1 [167..168)
 2
[201..202) 2,NumericLiteralExpression^ReturnStatement^IfStatement^Block_IfStatement_ReturnStatement_NumericLiteralExpression,1 [167..168)
 2
[191..192) 2,NumericLiteralExpression^EqualsExpression^IfStatement^Block_ReturnStatement_UnaryMinusExpression_NumericLiteralExpression,1 [225..226)
 2
[201..202) 2,NumericLiteralExpression^ReturnStatement^IfStatement^Block_ReturnStatement_UnaryMinusExpression_NumericLiteralExpression,1 [225..226)
 2
[191..192) 2,NumericLiteralExpression^EqualsExpression^IfStatement^Block^MethodDeclaration,METHOD_NAME [74..84)
 2
[201..202) 2,NumericLiteralExpression^ReturnStatement^IfStatement^Block^MethodDeclaration,METHOD_NAME [74..84)
 2

…ility function

urialon

Thanks, see minor comments

CSharpExtractor/CSharpExtractor/Extractor/Extractor.cs

CSharpExtractor/CSharpExtractor/Extractor/Utilities.cs

CSharpExtractor/CSharpExtractor/Extractor/Program.cs

urialon · 2019-01-09T09:41:31Z

Hi Alexey,
Thanks!
All changes seem to be correct and reasonable. Thank you for finding the duplication, it looks like a bug.
Please see minor comments

urialon · 2019-01-09T22:08:26Z

LGTM, thanks Alexey!

* Add sampling in GetInternalPaths * Add uniqueness check in Variables * Use StreamWriter instead of standard out * Add MaxContexts and output file parameters, add Reservoir sampling utility function * Flush stream to write last line * Pass ofile_name command line option to the python script * Change variable names to follow convention used * Specify ofile_name argument instead of stdout * Use a file pointed to by ofile_name rather than directing from stdout * Revert change in the Pool size * Use IDisposable to manage StreamWriter

ASvyatkovskiy added 5 commits January 7, 2019 11:43

Add sampling in GetInternalPaths

ca576f6

Add uniqueness check in Variables

f3aebfa

Use StreamWriter instead of standard out

269de07

Add MaxContexts and output file parameters, add Reservoir sampling ut…

39b7f1e

…ility function

Flush stream to write last line

f5f3fba

yahave requested a review from urialon January 8, 2019 11:21

urialon requested changes Jan 9, 2019

View reviewed changes

ASvyatkovskiy added 6 commits January 9, 2019 11:38

Pass ofile_name command line option to the python script

ba7174a

Change variable names to follow convention used

a909b15

Specify ofile_name argument instead of stdout

bd03dda

Use a file pointed to by ofile_name rather than directing from stdout

f7a4689

Revert change in the Pool size

ca6dcbf

Use IDisposable to manage StreamWriter

10ca08c

urialon approved these changes Jan 9, 2019

View reviewed changes

urialon merged commit 978564a into tech-srl:master Jan 9, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Duplicate path contexts in C# extractor #7

Duplicate path contexts in C# extractor #7

Uh oh!

ASvyatkovskiy commented Jan 7, 2019 •

edited

Loading

Uh oh!

urialon left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

urialon commented Jan 9, 2019

Uh oh!

urialon commented Jan 9, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Duplicate path contexts in C# extractor #7

Duplicate path contexts in C# extractor #7

Uh oh!

Conversation

ASvyatkovskiy commented Jan 7, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

urialon left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

urialon commented Jan 9, 2019

Uh oh!

urialon commented Jan 9, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ASvyatkovskiy commented Jan 7, 2019 •

edited

Loading